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ABSTRACT 

This report reviews the background of the 
Instructional Systems Design (ISD) model (used to develop training 
for Naval personnel), identifies problems in the ISD process and in 
its management and implementation, and recommends methods of ISD 
improvement. The ISD model is described as a process originally 
developed to remind instructional development experts about steps 
needed* to produce quality instruction, but subsequently implemented 
to help content specialists (who are relatively inexperienced in 
instructional design and development) build instruction. It is noted 
that ISD methods as used by nonexperts are not successful because 
they lack detailed procedural guidance. Instructional engineering and 
management problems in implementing ISD are outlined and three 
alternative solutions to these problems are considered and rejected. 
Several recent research efforts are then summarized, including the 
instructional quality inventory (IQI), which provides quality 
assurance methods for the ISD; the development of guidelines for 
building more relevant criterion-referenced tests; and the initial 
development of computer assisted training development. It is 
recommended that the Naval Education and Training Command 
(NAVEDTRACOM) develop: (1) systematic methods for monitoring ISD 
implementation and the performance of ISD practitioners and managers; 
(2) training and professional development programs for these persons; 
and (3) automated aids for ISD. A 36-item bibliography and a report 
distribution list are provided. (Author/ESR) 
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In :ias aaper, background developments leadir^g to the adoption of the Inslrii; l K)nai 
>ysteMis Developuient (ISD) model are described briefly. During the 1960s, a systems 
.'.nalvsis aDoro.iCh to training by a teanri of experts evolved into the ISD approach, which 
.issuined iriai lae rnetiiods used by er.perts could be described completely enougii ro 
nonexperts to earry out trainmg development. Recent studies evaluaiia,^ ISI"^ 
impie.mcntation suggest that this goal has not been reached. While ISD as a gener.ii 
^ curriculum deveiopn^ent procedure is adequate, it needs considerable detailed refinement. 
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arncle bu<riinar izing xhc sta:t^-oi -i;ie-ar l Ir. proc-jui.rc^ lor i.r.-i.niii,.^ ^'v-/ 
tlie role of NAVPERSR ANOCEN' research and develop! ner.i ir. aiisivciru; > 
. [^countered. It is intended to ;)rovide iTianpovv'er, personnel, anc; irainin;; n. 
general overviev.' of the backgroiirrl ei -nodern apprv^^.^cl^es x^^ iraii-.ini; Oevel 
out certain prooleins existing ir. prooediire'-. T!\^^vr>• ;V0:. iric 

nnplernentation, descrioe so:ne inodL-s; aLte;npLb to improve ;;.ip.<j-i,^^: 
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a:so or.-..ui^-- ' .'K :-L\vsing rec^inoi'^si'-i* ;:on'iplexi ly demands it. Fleet r"ac::^;e:;S 
..,;o!-^ pcrsoiiDol r<::jdiness, wuiv .ri :iirri. is iniiuerKcd by selection, u-ai:ij:i.;, joi 
v-xpenencc, inotiv _;tion, ieadersi.;p, and other lactors. To aciiiev.; j,'.;.vi:K.ia 1 
re'ad!P.e.ss--a;id, \n ine long run, improve unit and fleet readinesb--desi:^n lecnnoio-u-. 
nvabt be developed and applied in ail oi these areas. In training, a ucsign tcc:;w 
nology--lnbt--ictionai Systems Desi<4n (ISD)--has been developed and adoptee. Aitnoijgl: 

IS an .ACitmg prospeci for improving the quality of Navy training programs and ihi- 
n.anagefnent of Navy training developfnent, there are problems in its inanagemer,L ario 
i;i)jjio;Virntat:oii anci m the adequacy of ISD methods and procedures. L^ecause of this, ISlo 

no: ,ri acmeved its full potential for 'rr^proving Navy training. 

Y:\c purpose oi this report is ro review the Dackground of ISlj, ujentiiy proi:i'..;ins jri 
I'.ie [Sn ^;rocvrss and m us management and irnplementa tion, describe soine researcii and 
..cveiop ::eri: etrurts aimed at solving these problems, and recommend additiop»al ^e^<^iroh 
.re dec] . 



i:)eveiopments leading to adoptmg tiie model and systematic btudie^ (-/aiuatmg 
successes and problems of LSD implementation were extensively revicweri, hVofii 

tnese^ reviews, several research progranis were begun that aid m solving ccTtam ol tlv.: 

DrobkMf- ^ enountered. 

; ■ i I \(\ mg:-. an d Conc lusions 

Ti-.' ISn process is a switliesis oi :ne results of nearly ^^0 vear::> oi researcn 
. . :ruciionai development and over 1 • /; previous publications concernnig what to cio ui 
i;v.rru<,:tional development. Tlie review found widespread agreemient tliat tne ISi!) process 
i-, u good d<^scri[)tion of what has to be done in the process of analyzing, desigmng, 
•.eveloping, and managing instruction. 

The IST; methods were originally developed to reinind instructional deveiopmeni 
.'xp - :s ..bout steps needed to be accomplished to p-cduce quality instruction. Recently, 
rnoagh. the intent of ISD methods has shifted. ISD manuals are now intended to help 
, jni,cr>.i specialises (who are relatively inexperienced in instructional design and 
ievt-looment) build instruction. However, ISD metliods do not achieve tlas int<'nt, 
:j,;(:aiise. while they contain "what to do" loiormiation, they do not provide information :i 
■;u;w lO do 0.." 

Thi, lack i)i detailed procedural guidance leads to two difficulties in impiemer^nng 
instructional engineeniig problems and management probleuis. Studies tiave 
-..'vee.led tli^ii iS!^ methods are r-ot implemented well by the untrained, inexperle^■lr^•(.i 
^.^'tsoiinol availaoie. This leads, in turn, to proolen-is in managing their ISD a(tivitv. 
i"T; '/ proi:)lems result in the development of instruction imperlectly rnatf:hed to Navy ]ods 
af:c increasec costs for evaluation and revision of initjally poor instruction. 
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•\ number of policy or management rlianges to alleviate these probleri,:-, xere 
. onsidereci, including (1) substitution of some ottuiT training development methods in place 
01 ISn, ..•. US-ant:.; : -aimr;^ lor ISC pracutioiiers lo C'.-.'., . ..at.- iS."^ . .Hequa'— s 
and (3)' contracting for the d( 'ek^ment of Navy training programs. All these alternative.> 
were rejectc-d as bem^ unpractical, too expensive, or as simply cosmetic. Instead, ii was 
decided that the best policy to follow is to improve the ISD methods and tneir 
implementation t',-...'J.;^-i a ^y. .tic research arc dcveloprr.cn- pro---.:". S.-veral recent 
efforts were sun.,nari:v, >:.cluding th>, in:;-., u.iional quality inventory (IQl), whicn 
provides quality assc..; ■. diods for the I-^^H, the development of procedures for 

building more instructionaliy relevant tests m technical training, and the initial 
development of compu ;.;r-ba.-:-d systems to assi:it in managing and '^-^nducting LSD. 
Computer-based aids to ISD, i.. particular, : ; ..v :cc the best hope of improving the 
efficiency and effectiveness of the ISD process and are essential as more traming is 
delivered / '.a i, omputers. 

n . . .^^.,^^( 1 i '■ ,. c 

1 '., V./ 1 I 1 I t I v_ « < ^ ' i-i ' ■ / I . 

1. This review identified soine deficiencies in the ISD process and its implementa- 
tion. However, these def icie^ jes were obtrxined from only a few stndiei^ of 'SD and fmn 
rcla::v'.lv ;n:...-.nal observation. It is essential tnat the Naval Education and Training 
Command (N AVEDTR ACOM) develop systematic methods for monitoring the implementa- 
tion of LSD, so tiiat problems can be identified. These problems should then be reviewed 
periodically to determine their amenability to management, p^-rsonnel training, or R^D 
solutions. 

2. .\AVl:.DTRACOM should develop a system for mionitor ing the performance oi ISD 
r.ictitioners and managers and a systematic training and professional development 

. r.v-r<jL^ii to improve their performiance . 

NAVEDTl^ACOM should actively support the development of automated aids to 
.-•rs ;md should begin planning and programming resources for their eventual implementa- 
. . ■. .1 .d operation. 
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iNTRODUCTSON 



The \avy :v:'o:i > bc-itor ir-^ ' < '-.i not only because fleet recC'Tsesb nei,-?;.^' . 
al-n^ because incr(?asin^ :(^chnologi'":ai complexity demands ix.^ F'ieet reaoj'^^ss c'v ;viin'. 
apov. personnel readif^ess, wiuch. turn, is iniluenced by selection t : .i.r;..c..:.. 
experience^ motivation, leadership, and oti"^.er idctors. To acnieve indiNac^jc. ''^::.i->'- 
ness--and. in the long run, improve unit and fleet readiness--design tecnnolo.'^ies .;iJ': 
ceveloped .ind applied in . M of these areas. In training, a design technolo^y-^Ir strccr ion. . 
Sysl.ei!j:j Design (LSD)- -has been developed and adopted. Although ISD is ai; ex.ctm.-; 
prospect for improvin,^ the quality of Navy training programs and tne [nuna>.v--i ci!. .>f 
\:ivy trainmg development, there are problems m its management and implernr^r.toT lo-, 
and m the adequacy of ISD methods and procedures. Because of tr^s. ISI'!^ ri.:.^ nc: yei 
..i<iiie*V''-rl iis full potential for improving Navy training. 

Ob -c t!ve 

TfiC primary objective of this report is to review selected problems exisimv' in rhc 
i;:ipl<:' i'.*ntation of ISD methodology and to describe briefly some research and d^'velo;,- 
.neru ivrojects that are underway or needed to aid future implementation of the-[.^ro<:ess. 

APPROACH 

Developments leading lo adopting the ISD model and systematic s:,.":li("s evaiuaiir^^-^ 
::ie successes and problems of ISD^ implementation were extensivel\" revie^s^TK From 
tliese reviews, several research programs were begun t!iat aid in so! ■ i- g ce-'tain of tl\e 
problems enountered. 

REVIEW OF iSD 

Navy curriculum design efforts are continually being fiandicapped by sliorta^:,es of 
ex DO!" Is to conduc: training program design and development, poor analysis of how to 
n-iatcii training to jobs, inadequate prescriptions for deciding how to tram, inadequate 
performance measurements, differences in student skills and motivation, problems in 
inanaging Navy schools, and problems in planning tne use of computers and simulators lu 
training. It is important to recogni;:e thac these problems all stem from fundameiUal 
inadequacies in our understanding of how people learn and, therefore, how to teach these 
people. This leads to inadequacies in training design methods and procedures and to 
confusions about management of training program design and development. Researr.ii and 
development has led and will lead to improvements in our fundamental understanding of 
learning and instruction, and in procedural refmements that will lead to improvements In 
iSD methodology and the formulation of more effective management policies, r.areful 
implementation of existing knowledge and systematic development of information haS(*s 
for new procedures are needed. 



^Chjci of ^aval Operations Memorandum, Scr 00/100622 of ii Dc:ce:'^;be: 
CNO objectives for 1980. 
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;ie r:\oli.ition if ISO 



Traditional techniques for developing instructional programs depend very l.eavilN .... 
-V expertise of the people doing the development. I 'ntil recently, instructional de.:^!, 
■,vas an intuitive, artistic process. One proble-n w;;n : approach is that tne intu.i.ons ol 
Deople about what and how to teach .an vary widely. The resulting instruction may .ivon 
reach irrelevant things or nerhaps leave out thing, th.: are very important to Pe^P e 
performance. A second problem is tnat people's arti.try also varies, ^hile one artist may 
desien instruction that con.nunicates very efiiciently, another person may produce 
"instruction" that is barely c-unprehensible. Thus, instability or variaoility is built into 
the traditiona' instructional developmeni prorfss. 

This traditional approach has further problems. Although it is concerned with job 
r.^evancy in a general sense, it cannot ensure that training matches ]ob requirements: It 
might or might not, depending on someone's intuition. The traditional approach also 
cannot <msurt that qn^ 'y tr.fining is developed. Training materials may ^-eloped 
badlY or materials mav t>e chosen arbitrarily by an instructor or developer. Also, m the 
traditional philosophy .1 mst, .ction, the focus is usually on the content that is Presented 
to the trainee, wno is expected to learn from the exposure. Instruction is usually topic 
oriented" in that it tells about something (e.g., how r mdar operate) rather har, being 
"performance-oriented," which tells a student how to op^.rate tne radar, ^he learning i 
visually not evaluated systematically, and training adequacy is judged in terms of what 
students say about it. Little attempt is made to identify what difficulty a student is 
having in learning and to correct it. It is, in essence, a "sink or swim" approach. 

During the mid 1950s, training developers recognized the need to guarantee the job 
rHevancy and to monitor systematically inadequacies in learning. They began developing 
techniques to stabilize and structure the process of training development, to ensure tne 
relevancy of training for people's jobs, and to make training efficient. Th^^ approach, 
.dapted from those used in operations research and systems ^^^^''^-^^^^^^f ^[^^^^^^^ 
.>.SS) [or the development of weapon systems, led to the development of the ISD process. 

ISD procedures evolved from a conviction that th. systems analysis approach could 
simplify the complex task of developing, programs of instruction. Systems analysis 
methods had developed during World Wa. II to help resolve problems in managing the 
design production, and evaluation of new weapon systems. The method was apphed 
successfully to numerous problems whose complexity strained any one persons ability to 
comprehend and accomplish a task. For example, systems analysis has been given credit 
for the success of the project that put man on the moon (Carter, 

Four main procedures characterize the systems approach: 

1. A team of experts is chosen from relevant disciplines to bring as much 
information and expertise as possible to bear in accomplishing a task. 

2. Models or simplified descriptions of subparts of the task are generated to reduce 
its overall complexity. 

3. Unique but systematic solutions to the task are devised. 

k. Operational tests are conducted to provide information for later revision and 
modification (cf., Montemerlo & Tennyson, 1976; Andrews & Goodson, 1980). 

As applied to training program development, a group of experts in management, logistics 
education/training, systems planning, .nd other fields generated model procedures to 

ll 
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sirnplily day-:o-d..iy t:isk>. For ^.-xan ipi* * , t:.ii!iiii^ c\poris 'ie". ise ^; : i^-cK i i - : ^ :;r >>:!;-*r 

nines to r^-in.nu thcrri of siL^ps clove lopinent that had to bt.-" con iplotcd and to rc^ or.] 
^■■iCh ste_*ps when tliey had beer, cofnpleted. Althougli sucli procedures help expe-'is 
tj^'terrnine what to cio [ie>:t, ttiey oo ^\o'. supplant the intelligence or knowledge needed 
'*c:rry on the activny (W.-'ntemerlo v!/ I'erir; vsofu 1976; Monterntrrio, 1979a; And'^;; ws .v 

-j'->^d ^sOM, 1 

[during tiie i960s, tfu-re was a sluit aw^y iroiit thio systeiiiS analysis approacij, v.'hj';:; 
relied upon tearns oi experts, toward development of forinal procedures, models, ariL 
design decisio'^ aids that would enable relatively inexperienced persons to design instruc-- 
iion. These procedures and aids were elaborate forms of the simple models and checklists 
used by x\\c experts. The prospect of being able to use less experienced people to develop 
training appealed to managers of military instruction programs because experts were 
scarce--and still are--and job rotation restricted 'die buildup of expertise. Over 100 
inanuals were published telling how to design and develop programs of instruction (e.g., 
Moritemerlo Tennyson, 1976; Andrews ii Goodson, 1980). At various times, these 
procedures were called "course design procedures," "curriculum engineering," "systems 
approar.ti to training," and, m.ore recently, "instructional systems development." 

Mthough all these procedures differ in some details, they share a comirnon approach: 
T!iey analyze jobs to determine training objectives, develop tests to assess whether 
trainees are progressing toward objectives, gear instruction toward specific learning goals 
that are tied to the objective, and attempt to detemine how to decide upon th.e 
instructional presentations in sufficient detail to nunimize the level ox experience needed 
m instructional development and technology. 

ISD evolved not as a specific method of teaching but, rather, as a way to determine 
what trainees really need to !<now and to ensure that they learri it. With reference to 
training, ISO's goals are to make training (1) job-relevant (meaning it would ultimately 
prepare the trainee for his or her function(s) in Navy readiness), and (2) cost-effective and 
efficient (meaning it would use the miost efficient training methods to do the training). 
With regard to the management of the development of instructional programs, ISD's goals 
are to (1) make the process more efficient and less haphazard and (2) provide a basis for 
(rontroliing and evaluating changes. 

It is important to understand these training and management goals because people 
liave often confused ISD with particular new teaching techniques. The ISD process is a 
means for defining training goals, deciding upon the best means of achieving them within 
resource constraints, and providing evaluation of the program. In theory, any medium of 
presentation (traditional, self-paced, or computer-managed) could be chosen, depending 
upon its feasibility. In practice, however, the recommended method is self-pacing 
because it is more efficient. Self -pacing takes advantage of the fact that people learn at 
different rates (Branson, Rayner, Cox, Furman, King, & Harnum, 1975). Research has 
shown that, when self-paced courses are compared with traditional "lock-step" courses, 
they save considerable time, with students achieving the same or slightly better school 
performance. For example, in comparisons done in ^8 military training courses, 
achievement in self-paced courses was equal to that of conventional courses in 32 cases, 
superior in \ 5 cases, and sligntiy poorer in only 1 case (Orlansky & String, 1979). There 
are a number of questions- -v/hich research can help answer- -about how to implement 
other types of individualized instruction and whether they are effective for all trainees. 
Nevertheless, "its use can be expected to continue and expand in the Navy as increased 
emphasis is placed on training efficiency," according to a 1979 report by the Navy 
Training Analysis and Evaluation Group (Zajkowski, Heidt, Corey, Merv, & Micheli, 1979), 
unless it can be shown that traditional methods can be made more efficient through the 
'ise of innovative techniques, such as computer-aids for instructors. 
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iSn Poj.e(UKils_aiH.i Pr.>hh:MirN 

History shows that t'doc:atiofv:il iiii iovath)P.s ui liUS century go through a p-.- /uhar 
three-stage hie cycle (Moiitciiierlo .V T.-uv^son, \97h: Campbell, i97 1). In tlie first stage, 
advocates'' of an Inriovation prociaun ;t> usefulness and its suc^.ess. In stage two, rnan.y 
people are attracted to the innovation and begin usinr, it. The final stage, however, is one 
ol ^growing skepticism and criticisfn oi ihc in:.ov.i;;on'> adequacy. :v:'cause r!us criii<asm 
coines late in the process, it does not help impr-.^ve the techt^ique bat, rathoc, hastens its 
abandonment in favor of yet another innovation. Ttie process then begins anew. This may 
explain why so many training development systen)S differ only in slight degree from one 
another (cf., Montemerlo lV Tennyson, 1976; Andrews Goodso'i, 19S0). 

Whatever the causes of the cycle-~and sofne tlx-.jrisis believe they are political and 
s.)cial (Milsum, 1968; Montefnerlo, 1979b)--the effect is that, when an educational 
maovation is introduced, its proponents suppress constructive criucism as they nurture 
afid protect their "brain child." The danger is that ihe ISO model, which has enjoyed ns 
days of advor\icy and is now somewhere between phase two of widespread use and phase 
three of growing skepticism, may share the fate of so many other innovations. Yet any 
systefii that replaces ISD will present the same underlying proDierns. it will bc! new, 
unrefined, untested, and probably difficult to inanage. 



As LSD moves into the criticism pliase •){ its liie cycle, it raiH retains (ruich of its 
early prosnise. ISO has made progress m developing techniques to make ti aining more job- 
relevant. As will be discussed below, liowever, problems exist in successfully implement- 
ing these techniques. The fact that some verMon of the systems approach model has been 
adopted by the military services, by industry, and i.>r traii^ir^g development in the civilian 
sector suggests that the approach is valid (Cropper, 19S0). This has happened because 
systems approaches all include the same general steps that serve as a basis for managing 
the process, and managers believe that systematic attempts to relate training to job 
requirements optimize the use of resources better than relatively unsystematic alterna- 
tives. The original hope that the procedures of experts could somehow be transmitted by 
means of manuals to nonexperts, who then carry out training development, does not seem 
to nave been realized (\Aontemerlo ^ Tennyson, 1976; Montemerlo, 1979a). This failure 
stems from a less-than^adequate state of knowledge ah ^ut human learning and instruction, 
as well as our inability to provide recipes for training development that untrained people 
can follow. Therefore, the difficulties encountered in using the LSD process are of two 
sorts: "instructional engineering" and "n .ciagement" proL.-ns. These problems are 
dr- .issed in the following paragraphs. 

Instruc tiona l Eiigmeer ing Proble ms 

Current LSD procedure manuals are supposedJv more com[)iete tl^an earlier ones and 
more relevant to the development A training for a wide variety of military tasks. Yet, a 
recent study of the implementation of 150 by 33 groups that developed r/ different 
rourses in all the iralitary services noted that the procedures are still not fully adequate, 
in this study, Vineberg and Joyner (1980) reported that the job-relevancy issue was often 
ignored and that previously existing instruction was used as a starting point for course 
(i-velopment. Instructional methods were selected not because they were effective and 

-.a.^rt but because they existed. Similarly, tests to measure job-related learning were 
ill:. -ted to v'hat could rather than what should be tested. Evaluation of trainm , according 
TO Vmeb^rg and joyner, received little emphasis. Feedback systems from operational 
:][;CiS concerning joD competency of graduates were not well developed. a result, 

^j,:. .rograrns ^.-.eded extensive tryocts and revision to make them effect. ve. These 
nndings suggest that, at present, implementatior. LSD has not succeeded m attaining 




t:ie ^;oals of :naki!ig trdiiung job-relevant, efficient, or Ck)St-eflecTive. LvKic^:, • tr.is 
has been given in reports of studies of the nnplernentation of systcMis appro. ici-i^\-. in 
trainii^.g development. Army users, for example, have found guidance incomplete, steps 
difficult to relate to one another, and job analyses incomplete (Vineberg 6c joyner, 1980; 
Ricketson, Schultz, ^ Wright, 1970; Hodak, Middleton, & Rankin, 1979; Miller, Swink, A 
\KKenzie, 1978). These findings suggest that those procedures were not implerruMned 
well bv the untrained, inexperienceo personnel available (Montemerlo A: Tennyson, h 76). 

Our analysis of some Navy courses reveals similar serious deficiencies in traini:rg 
objectives, tests, and course materials (e.g., Wulfeck, Stern, F-'^dericks, Ac r i^, 1979, 
Stern A: Ff^de-irks, in press). Some objectives are not related to perfo. Dance or 
knowled^^;..' icc; :.red by the job. Testing does not always measure performance or 
knowledge required by the objectives. Since mstruction often is not geared to the 
.objectives or to the tests, it is confusing and otherwise inadequate. These problems 'nave 
ic'c, in some cases, to course graduates who are poorly prepared for their jobs, and th.s ha- 
resiilt^'d in criticism of ISD by the operational community. 

Mow can it b<^ that these problems result even though tne ISD procedures are 
. ..oscdly (nore complete than earlier instructional design systems? It appears that these 
:.tru(:tionai engineering problems are the result of the fact that the ISD procedures 
pecifv "what to do" in detail, but not "how to do it," let alone "how to do it well." 

\[ m:gin. be us<:iul to underscore this poiQl, For someone who is very knowledgeable 
.i:v.: .killed in a particular discipline, a simple reminck;r that sometiiing should be done ;nay 
suftMiient. ThK> why simple checklists worked well in the early attempts to use a 
-^vsceir.s approach to training; development. In the development of procedural guidehnes, 
such as those in ISD, the adequacy of the guidance for inexperienced people is the issue- 
Murk; .-nore specific "how to do it" information must be provided. 

An interesting analogue of this problem was discussed ir^ a recent paper. In a 
■: .-.okbook meant for experienced chefs, a recipe for mayonnaise was found to i^e six 
:,--;.:-nces long (plus a list of ingredients). In another book, intended for less experienced 
'.he same recipe was over 13 times Ionger--sonie 900 words plus ingredients 
(Norfnan, Geinner, A: Stevens, 1976). Even with this recipe, a complete novice in the 
kitchen would have difficulty in completing it successfully. 

Are instructional recipes any different? Their adequacy depends on tiow w(m1 the 
persons doing the work understand what has to be done and all the steps in the process. 
To Vnow tne difficulty with the level of the "recipes" presented in ISD, consider that, out 
of a 36-page section on developing instruction, only two paragraphs~-four sentences--tell 
-ow to develop printed training materials, and only one of those sentences offers real 
j;jidance: "Write all the student needs to know about each learning objective, but do not 
vnte one wi>rd more " (Branson et al., 1975, Vol. Ill, p. 2^6). Does this provide sufficient 
iidance for a person inexperienced in techniques of instruction and communication" 

in addition to the lack of adequate "how to ao it" information, the ISD guidance also 
.^r^ntams internal inconsistencies. For example, the ISD process requires that developerb 
specify objectives, (2) develop tests, (3) sequence the objectives into modules, lessons, 
.;tc., and' (^j develop instruction. The problem is that, although test itteiis can. be 
:;enerated from objectives, they cannot be arranged into modules or lesson te-.ts -ntil the 
>iDjectives have been set;, lenced. Therefore, things are out of order. The gu:delines say to 
-onstruct tests before we know what they should contain. Similar inconsisiencies can be 
Lound througiiout the ISD manuals. 



Manag e ment Problems 



These fundamental problems in the adequacy of ISD guidance lead toj^roblems u^ 
managing the ISD process. The management problems are further complicated by the tact 
that managers may not be aware of them. Training development managers often treat the 
ISD model and procedures as if they were complete and adequate and assume that, if a 
developer merely follows '^the book," mstruction will be adequate. There is no -ecogniticn 
of the variability caused by the inadequate procedures. Since the ISD model contains no 
methods for assuring the quality of materials as they are developed, managers may pay 
little or no attention to that quality. They do not know how. Rather, they consider the 
development of training to resemble assembly-line production; thus, the documentation of 
each step in the process receives more emphasis than the way the inadequate guidelines 
affect instructional quality. This trivialization of the process can and does result in 
inadequate programs that need numerous revisions to make them job-relevant. Managers 
tend to be satisfied when all tasks and forms are completed, even though students may 
falter (e.g., Montemerlo, 1979a; Vineberg & 3oyner, 1980; Hodak, Middleton, & Rankin, 
1979; Miller, Swink, & McKenzie, 1978; Middleton, Zajkowski, & Aagard, 1979). 

Other more apparent management problems include selecting and training design and 
development personnel, financing, and other administrative and logistics issues. In the 
area of selecting and training personnel, the failure to spell out procedures in sufficient 
detail makes it necessary to find or extensively train people with expertise in training 
technology. Costs are further driven up as each new instructional program requires (1) 
new people to learn how lo develop it and (2) extent. ve revisions to validate it. The 
decisiOii bv the Naval Education and Training Command (NAVEDTRACOM) to establiish a 
central organization for developing training programs reflects a tacit recognition that 
some expertise is needed to compensate for incomplete procedures. The Navy's 
Instructional Program Development Centers (IPDCs) include civilian technologists who are 
expected to have or develop and maintain ex^.-rtise. Most course development and 
revision will take place outside these centers for some time, however. Building up 
expertise for regular duty Navy personnel is difficult because of job rotation policies and 
: he lack of appropriate occupational specialties in instructional technology. Navy people 
.:o provide subject-matter expertise and, despite their lack of appreciable training in 
instructional development per se, are given responsibility for developing advanced 
courses. In this case, the quality of the instruction depends on the intuitions of the 
personnel. It is not necessarUy bad, but its quality is uncertain. 

It has recently become clear that the costs of adopting the ISD model are high. 
However, instructional development costs may always have been high . !n the past, these 
costs were "hidden" in the normal assignment of instructors to Navy schools. To illustrate 
this point, in the Navy and in the other services, 25 percent of an instructor's workweek is 
accounted for by categorizing it as course preparation or administrative duties. There- 
fore, traditionally, course development or preparation costs are high even if only half that 
time is considered to be course development. In addition, actual course development was 
and still is done within Navy schools. In a survey completed several years ago, it was 
found that, for a 2 1/2 year period, four to seven instructors had been assigned to develop 
objectives for a new course. They were considered part of the school's complement of 
instructors. The adoption of the IPD approach has made costs more apparent because of 
:he way in which costs are reported, and not necessarily because costs actually have been 
increased . ^' ^ 




^Chief of Naval Education and Training Instruction 1550.5 of 11 March 197^f; subj: 
Doctrine for the centralized management and control of ISD. 

^Chief of Naval Education and Training Support Code N^, Direct Cost Report, of 30 
3une 1979. -f - 
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APPROACHES FOR IMPROVING THE LSD MODEL AND iMPLEMEN i A i h 



[ ^olicy Cons ide rat ions 

As wc have seen, lundanieiit.il instructi^^ruil engineering, problenis ;n IS!) * reate 
problems in the i!n[)le[nentation afvi ! lUinagernent of instructional deveiopnu^ir^ pro.;rar;;s 
in the islavy. What policy alternative^> are available to solve these probh.NV.s'' 

One alternative could be to discard the current ISD methodology and "start trom 
scratch." As stated earlier, however, all systematic approaches to instructior\al develop- 
ment include the same general steps. As a "what-to-do" procedure, ISO is generally 
adequate, and anything that replaced it would be fundamentally the same. 

A second alternative would be to leave ISD as it is but increase the training of ISD 
practitioners so that they could compensate for the inadequacies in the process, it would 
be necessary to determine the amount of training developers need and to provide it on .i 
. ontinuing basis for both Navy civilian and military personnel. N AVEDTR ACOM's policy 
of developing courses in IPDCs is intended to build up the level of expertise. These 
centers, however, concentrate on preliminary technical training, and apprc. t ice-level 
courses, and their expertise will not generally be available to the personnel developing and 
■ nodifymg the thousands of other Navy courses. Wider dispersion of training and otUct 
:neans of assisting development of these coui\se> would be needed. There are some serious 
hm:tatio[is in giving training in instructional technology as a solution to this problem. 
Sinc(f tlie knowledge base for the ISD procedures is incomplete, training for ISD 
practitioners would be difficult to specify and develop. It would probably have to be 
broad in scope and lengthy--probably equival'^nt to a graduate prograin in educational 
technology--and would have to U'^ offered on a continuing basis because of personnel 
turnover. The impracticality and expense of such a program is obvious. Naturally, under 
any conditions, some in-service training will have to be offered on a conti^nimg basis, and 
system.atic evaluation of people's performance is necessary to assess their skill and to 
maintain their competency. Such training and evaluation, however, will not be sufficient 
;o compensate for ISD inadequacies. 

Another alternative is for the Navy to abandon the curriculum developmer^t business 
completely and contract for the development of training programs. This approach has 
serious limitations since potential contractor^) (I) have no better procedures for course 
.! -.ign than the Navy does, (2) are not sufficiently familiar with Navy operational and 
training situations to develop courses that fully meet Navy requirements, 0) tnay not 
exist in sufficient nufnbers to handle the c: irse develop :ient requirements, and M may 
cause irnplementation problems resulting from the "hand-off" process from contractor to 
Navy personnel. 

Tne policy alternative that seem^ ;nost appropriate is to retain current ISD 
nethodology in its general form, but to actively support a program of research and 
development aimed at refining it, augmenting it where necessary, and improving its 
impleinentation and management. The need for systematic refinement of ISD methods 
and their implementation has already been recognized. The Chief of Naval Bduc.^.tion and 
Training and N AVPERSR ANDCEN have agreed to identify and seek R&D solutions to 
problems that impede successful implementation of the ISD process.'' 



\Memorandum of Agreement between NAVPERSRANDCEN and CNET on researcii 
requirements, im.plementing R&D, and establishing the Experimental Trainini; Programs 
Policy Board, ser 28^, 13 May 1975. 
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Iinprovin)^: ISP Through Research and Development 



The main deficiency of ISD procedures discussed so far is their failure to allow 
relatively inexperienced personnel to develop reliably good instruction (Hodak, Middleton, 
& Rankin, 1979; Miller, Swink, Ac McKenzie, 1978). This section describes some research 
and development efforts designed to refine and augment the ISD methodology through (1) 
conducting R&D aimed at acquiring the knowledge base necessary to develop more usable 
and manageable ISD techniques, (2) improving the current process by developing 
procedures to fill ISD gaps (e.g., quality control techniques and test development 
inethodologies), and (3) improving the implementation and management of ISD by 
capitalizing on computer technology. 

Th e Instructional Quality Inventory (IQl) 

After examining the ISD process, it was recognized that, at several points in the 
developmental sequence, intermediate products (e.g., objectives, test items, segments of 
instruction, etc.) were available, but there was no check on their quality. This is 
particularly unfortunate, since later steps in the ISD sequence depend on the quality of 
these products. Therefore, the development of techniques for quality assurance during 
ISD was undertaken. First, the research literature was examined to determine what the 
tested or valid principles for prescribing instruction were. Then, techniques were devised 
to allow an evaluator to examine instruction and determine whether or not it conformed 
to those prescriptions. In brief, the evaluator makes sure that (1) the stated objectives 
are job-relevant, (2) test items are adequate for the purpose of assessing learning and are 
congruent with the objectives, and (3) course content is matched appropriately to the 
objectives and thereby to the job requirements. This technique is called the Instructional 
Quality Inventory (IQI). 

A preliminary version ol the IQI was developed by Professor M. D. Merrill and 
dissociates under contract to NAVPERSRANDCEN (Merrill <5c Boutwell, 1973; Merrill Ac 
Wood, 197^; Merrill, Richards, Schmidt, Ac Wood, 1977). The IQI then underwent extensive 
testing and revision in Navy courses to ensure that the procedures were clear and could be 
used by those who would be doing instructional development. User manuals, training 
materials, and job-aids were published (Wulfeck, Ellis, Richards, Merrill, Ac Wood, 1978; 
Ellis, Wulfeck, Ac Fredericks, 1979; Ellis Ac Wulfeck, 1978). The Chief of Naval ^Education 
and Training has since recommended their use throughout the NAVEDTRACOM. ' 

In helping make instructional materials and tests consistent with learning objectives, 
the (QI fills important gaps in ISD procedures that were revealed in surveys of 
instructional development personnel and of ISD implementation (Vineberg Ac 3oyner, 1980; 
Hodak et al, 1979; Middleton et al, 1979). Until now, no systematic means existed for 
determining whether existing instruction is adequate for current purposes, thereby 
avoiding unnecessary development costs. ISD did require that' existing instruction be 
reviewed bur provided no guidance for this. As a result, this step received little attention 
tn the development of new courses (Vineberg Ac 3oyner, 1980). The IQI procedure can also 
be used to examine the quality of contractor-developed instruction. 



'Cl.iof of Navc. Education and Training, CNETNOTE 1550 of 6 April 1979; sub]: 



[ istruotional Quality Inventory (IQI). 

^Zhicf of Naval Education and Training, CNETINST 1550.15 of 29 May 1980; subj 
Instructional Program Developm.ent Centers (iPDCs); policy and guidance for. 
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Ciuideltnes for_ Cri tenon-re (erenced Tc-.t in^; 

One of the goals of ISO is to foake training relevant to job-perfor;i^aix- e r^^qiiire- 
•nents. Such performance-based training requires techniques for testing diffc^re^ii iroin 
those familiar to most people. The tests commonly encountered are those t[iat are used 
to compare a person's test perforrnance with that of otiiers. Examples of such tests are 
those used to select people for admission to training, advancement-in-rate, or college 
entrance, or to rank people relative to some group as in IQ tests, Civil Service 
professional administration career examinations, and the Armed Forces Qualification 
Test. These tests are called "norm-referenced," meaning that an individual's test 
perforinance is compared--referenced~-to the average or "norm" of everyone else's test 
scores. Normative tests are, at best, only indirectly related to people's jobs. Iri coiUrast, 
performance-based training requires tests that directly measure aspects of performance 
required in course objectives. Tests of this type are called "criterion-referenced," 
meaning that an individual's test performance is compared--refer^nced~-to an absolute 
objective or criterion indicating what a person must know or be able to do. Quite 
different problems cir- confronted in developing and using these tests in cofUrast tv) norm- 
referenced tests. 

In addition to determining tliat an individual can do or does know whatever is 
, [decided in thr objectives, it is necessary to be able to determine why a trainee cannot 
i)erform well so that the problem or errors can be corrected. Whenever remedies are 
prescribed on the basis of test results, diag nostic tests are needed . Diagnostic tests must 
t)e designed to give further information about gaps in student knowledge or skill, i-.st as a 
p[iysician uses a^dditional diagnostic tests when symptoms indicate a medical problen^. For 
example, cof^sider a situation in which a student does not understand some ii^structiofi he 
has just read concerning when to use a particular pro-. -dure. When tested regarding whei^ 
ro use it, he makes an error. Since this is the only information the instructor has, tfie 
cause of the error is unknown. Usually, the student is given the same material to reread, 
if the misunderstanding is due to the fact that the explanation is unclear to the student, 
many rereadings of the instruction will not correct the error. Techniques for identifying 
why students make such errors are diagnostic. Once the misunderstanding is diagnosed, it 
an be corrected quickly. 

Although both criterion-referenced and diagnostic tests are needed in ISD, guidelines 
are incomplete for developing the former and nonexistent for developing the latter. 
Therefore, improved procedures for test development in ISD are being deveio[)ed. The 
'-esulting "handbook" will compile procedures from diverse sources (Merrill Ac W'ood, 197^; 
Swezey Pearlstein, 197^; Roid & Haladyna, 1979; Frederickson, Smith, A Pearlstein, 
1979) and arrange them as "job-aids" for Navy test developers. The handbook is currently 
being refined during training workshops for Navy personnel and a final version will be 
provided to NAVEDTRACOM in FY 1982. The handbook will provide better guidance for 
test development throughout the NAVEDTRACOM. It will provide more complete and 
;nore usable procedures than now exist. 

^ ' nputer-assiste d Training Development 

The earlier description of the ISD process gave no indication of record-keeping 
re: aire nents, aitliough formidable record-keeping problems exist. A typical Navy 
irainm^ program has hundreds or sometimes even thousands of learning objectives tl^at 
rnust oe developed, cross-referenced, tested, and taught. For example, about 7,000 
learning objectives are contained in the training for P-3 aircraft crews (Daubek, 
Freedrnan, WalKer, & Thode, in press). 
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Records also cover a wide variety of otiier bD a tivitiss such as generating test 
items, choosing alternative training media .md strategies, evaluating graduates, and 
revising courses. 

Computerized data management systems not only can assist with these record- 
keeping problems but also facilitate the development process itself by providing guidance 
for test and instructional development and similar tasks. Moreover, computer-based 
systems can ensure that guidance is followed by monitoring and evaluating developers' 
performance and by assisting them as they proceed. Computer systems can also provide 
training for instructional developers, who can fit it into their work schedules. 

NAVPERSRANDCEN is now developing computer-based aids for instructional design 
and development and will evaluate their usefulness. The first phase will help designers at 
critical points in the ISD process by guiding and monitoring each step involved in using the 
IQl and developing tests. Then programs will be expanded gradually to provide specific 
guidance on accomplishinn each task, allcv access to relevant data bases (e.g., test-item 
files, classifications of objectives, etc.;, and help select alternative forms of instruction. 
These aids will include computerized ISD data management. 

Some computer-based aids will be adapted from those being developed by other 
agencies. For example, the Army Research Institute is developing aids to help authors 
write, edit, and produce training materials (Schultz, Hibbits, Wagner, & Seidel, 1979). The 
intent is to make the production of materials simpler, faster, and less costly. Another 
system developed by the Navy Training Analysis and Evaluation Group (TAEG) (Braby, 
1979; Braby, Parrish, Guitard, & Aagard, 1978) can be used by authors who are subject- 
matter specialists to write, edit, and produce programmed training materials. The 
authors use a terminal that is connected to an editing and production system. An 
executive program requests information about materials and test items that the author 
provides by typing on a keyboard. The system then arranges the material in a particular 
format and produces a programmed text. Time required for authoring and producing 
materials is reduced and, more important, requirements for instructional design expertise 
are reduced. Such devices enable personnel with only modest experience in instruction to 
develop quality instructional materials. 

TAEG has also developed a computer readability and editing system (CRES), which is 
designed to improve the ease of comprehending Navy technical manuals and training 
materials. The system has features to detect uncommon and misspelled words and long 
sentences, suggest simple replacements for difficult words, and calculate the readability 
grade level. Each feature is consistent with Navy specifications and has been tested to 
verify that it provides useful feedback to editors and authors (Kinca-d, Aagard, & O'Hara, 

1980) . CRES has recently been extended to incorporate some of the IQI criteria, so that 
it assists authors in building better tests in technical training (Kincaid, Braby, & Wulfeck, 

1981) . All of these systems are prototypes of more advanced ones that wili design as well 
as produce instructional materials. 

The systems described so far are intended mainly to support development of 
conventional training materials. However, as most people are aware, there is a current 
revolution in computer technology for training delivery. Computers are getting smaller, 
more powerful, and much less expensive. There is also a revolution in the areas of 
artificial intelligence and cognitive science that is less visible but perhaps more excttmg. 
New techniques are being developed for giving computers real knowledge, for interacting 
with students in a more tutorial fashion, and for improving the interface between the 
student and the instructional device. These developments have the potential for making 
training better, cheaper, more manageable, and more widely distributed. 
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As attr'aL.uve wis liiesr (tev{:*u:)p; i;c!! is .un-, ii : aniK)i bi- .iShitincf; li^i ; '.vJi 
inuviediaiely revolutionize traiiunp. ^^Airrently. the ISO fiianuah <:oi\tain h.riioi 
rvMicerning cornputer-basecl mstr tional delivery. Yet, if saci^ devic'es art- aul dy 

i.ised, iuetiiods lor de.->ii^!niii^; arul vk-veloping instructional soitu'are inusi n(; 
ThMialized. Tlio introductiof) oi ;A)rni;uiers into thie \SD jj^rocess (.:omplJCat«'.> iu (;'^( aus- 
cidditional aitefition niust or ^^iv.-n during tlu^ analysis arul cteveKipinent p.kis'/\. 
instructional logic (whi< !^ iiistr\i< iv;,r noririally does), pianiiing tiic siu^!<.:i : : ^ 'i i i;^^vCif i 
interaction arid interlace, tlu: tv,H;s ^--t student rrs;)>.')[ise data requirr-d. tlu- ' i.^- >li iOCr • 
utilization and n'iaunain..tl)ili i\ cl lik- h..cdwarc aiKi Sv)l t\v,'arc, afic .1 '. .a'h Cs ' C :!:(■; 
i',su(;s. C\)nipu ter-ba.si'd author aui^i, ti'HT., must ix- ;a'Vcloped ir) support i^i^) a/ a "yp^'" 
*)t i.i^.truc ii(>nal delivery. 



CONCLUSIONS 

pro(X\>^s IS a syiuhesi^ ot tihc results of f:rarly ye-ars oi i^'\'.'-ral, '.m: 
m^truc ti^)[Ml development and over ]00 previous publications conrerning wdf lo -io >: 
in^tru< tional development. Tlic review \n\\nc\ widespread agreement that the {.u 
!s a t'.ood description i.^l wliat lias ro be done in the process ot analyzin,.., dcM..: niiig . 
■:rV( lo;)ing, and mLinaging instrui" t ic-i 

fne ISl.) metiiods were origmaiiv aeveioped to reimnd in:>tru(^ti^HCii ( i' a -| -1 ; s . . 
t/xperts about steps needed to be accomplished to produce quality instruc tioic lvo>a itiiv. 
tnougn, thie intent of ISD methods i^as stutted. ISD manuals are wow micridrd 10. ii>-!p 
content s|.:»ecia lists (who are relatively inexper iefu:ed in ir,structio[ial uemgn .litd 
development) build instructioii. However, iSD methods cio not acliieve tm- icteiiL, 
oecause, while they contain "wliat to do" iniormation, they do not provide inior laai k^u on 
"now to do it." 

Tlus lack oi detailed procedural guidance leads to two diilieulties m iiap'l< •mem = 
ISH: instructional engir^eering problems and management problems. Mudies aav ■ 
revealed that ISD methods are no; implemented well by tiie untrained, inc^xperii^r^c- i 
..ersonnel available. This leads, m turn, to problems in managing thieir l^D acii\c > 
I'nese problen^s result in the developnion t instructioii imperfectly matcfied tc \av\ ,oi"jN 
afid increased costs tor evaluation and revision of initially [)oor mstructioru 

A fiuiriber ot policy or nKinageiiiciu eC;anges 10 all<"vuiTe Ciese . .vie 

considered, including (1/ substitution of some other traii^ang deve!- aeni [neiliod'. ri.v- :■ 
ot ISD. (2) suostantial training for ISD practitioners to compensate for ISO maC , qi..i- k:-.. 
.iiic (3) contracting for the development of Navy training programs. Ail tiiese ..iii-r^i aic/'-^ 
were rejec:ted as beii^g impractical, too expensive, or as simply cosmetize instead, it \\\i-> 
uecided that trie best policy to follow is to improve the ISD metliod:. aiic mer" 
implementation through a systematic research and development program. SovtM\o r<'(^ent 
.vlM^ eifo.'-ts were summarized, including the IQI, whicii provides quality aNsur.>ric c 
■i-.etfu^as lor the ISD, the development of procedures for building more instru<:Tc)nai:V 
rc-.ewLru tests in technical training, an.d the initial development of t omputei-Dasec 
v. -,rv;i:^. to a^h.st m managing and conducting ISD. C'^o-; iputer-based aiOs to i^D. .ri 
,..i:c .■ "elar, jfov.de the best hope of improving riie elficiencv and et feet iver^ess -^1 [■'.'■■ \^^\) 
r .1 Lir-e e^^^erilial as rr.ore training is delivered via ei:>mj,>uters. 
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RECOMMENDATIONS 



1. This review identified ^ioine deficiencies in the ISD process and its iip-plcinenta- 
lion. However, these deficiencies were obtained from only a few studies of ISH and froiii 
relatively informal observation. It is essential that NAVEDTRACOM develop systematic 
methods for monitoring the implementation of ISD, so that problems can be identified. 
These problems should" then be reviewed periodically to determine their arnenabihty to 
fnanagemenl, personnel training, or R6^D solutions. 

2. NAVEDTRACOM should develop a system for monitoring the performance of ISD 
practitioiK^rs and managers and a systematic training and professional development 
prografn to improve their performance. 

3. NAVEDTRACOM should actively support the development of automated aids to 
ISD and should begin planning and programming resources for their eventual implementa- 
tion and operation. 
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